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DETAILED ACTION 

1 . Claims 1 -43 are amended. 

Claims 1-43 are presented for examination have been examined on merits and 
are pending in this application. 

Response to Amendment 

2. Acknowledgment is made for Applicants Amendments for claims filed on 
06/30/2008. 

As regards to IDS forms PTO-1449, it is not necessary to initial the respective 
boxes in the PTO-1449 forms, since all references were considered except 
where lined through and initialed at the end of the form by the examiner. 
Claims 17-20 rejection under 35 USC § 101 is withdrawn as Claims 17-20 are 
amended. 

Applicant's amendment necessitated updating search and new ground(s) of 
rejection presented in this office action. Applicant's arguments are deemed 
moot in view of the following new grounds of rejection as explained here below, 
necessitated by Applicant's substantial amendment (i.e., reference information 
receiving means that receives reference information from the one or more user 
devices, the reference information specifying contents to be referred to by the 
related information and specifying a data reference position in the contents ) to 
the claims which significantly affected the scope thereof. 
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Applicant's arguments are deemed moot in view of the following new grounds of 
rejection as explained here below. 

Applicant's arguments with respect to amended Claims 1-43, have been 
considered but are moot in view of the new ground(s) of rejection. 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

3. Claims 1-43 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
U.S. Patent No. 6493872 to Rangan, P. Venkat et al., (hereinafter "Rangan") as applied 
to Claims 1-43, and further in view of US Patent No. 7143428 to Bruck; Timo et al., 
(herein after "Bruck"). 

a. Regarding Claims 1-43 Rangan discloses substantially the invention as 
claimed. Rangan does not explicitly disclose wherein a related information 
receiver that receives related information. 

b. Rangan discloses (as stated in col. 5, lines 64-67, col. 6, lines 1-5, 
authoring station 11 adapted for accepting a video data stream 16 from a video 
head end 15 as is known in the art. The original video data source 12 may be 
any recorded or live source and is not limited as to protocol. 
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c. Rangan does not disclose that the authoring station 11 adapted for 
accepting a video data stream 16 from a video head end is related information on 
contents comes from one or more user devices. 

d. Bruck does disclose, (as stated in col. 4, lines 13-14, lines 33-67, col. 5, 
lines 1-2, col. 7, lines 26-58, FIG. 2 is a block diagram illustrating internal 
features of set-top box 12. A receiver 44 receives broadcast video signals, such 
as an NTSC video signal. The video signal may be received and decoded directly 
by the receiver or may be a feed from a separate box, such as a cable decoder 
or a satellite receiver (not shown). Typically, the receiver is able to tune into a 
selectable channel or control/monitor the tuning of an external device such as the 
satellite receiver. This capability allows the set-top box to know what video signal 
is being processed, which in turn enables various functional capabilities 
described herein below. The receiver is coupled to ASIC 32 to provide 
corresponding electrical signals to ASIC 32. This allows video data carried in 
broadcast video signal 20 to be presented to the viewer on television 14. More 
specifically, receiver 44 is configured to digitize the incoming video signal, which 
is stored in RAM 48, so that ASIC 32 and CPU 30 can process the video signal 
before displaying it on the television. This configuration enables the set-top box 
to generate shared screen viewing where the video signal only occupies a 
portion of the screen. The remaining screen area then can be used to display 
various user interface controls or data, as described in more detail below. By 
virtue of the combined video and Internet browsing capabilities of the set-top box, 
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it is possible to provide functional integration between television viewing and 
Internet browsing. For instance, one feature of set-top boxes, as described 
herein, is provision of an electronic programming guide for television viewing. 
The electronic programming guide is an electronic database on the client which 
is periodically updated by the host server. The database includes scheduling, 
rating and other information about the television shows available to the user. 
Among other things, the electronic programming guide also can include links to 
websites associated with various shows. Using information associated with the 
programming guide, users are, for instance, able to conveniently access 
websites that relate to television shows using on screen links. Upon activation of 
the link, the user is transferred into a chat room related to the particular television 
show being viewed as depicted in FIG. 6. In the chat room of FIG. 6, text 
messages from the new user (as well as other participants in the chat room) are 
displayed in a chat region 108 of a chat user interface 110, as symbolized by the 
large "Chat Transcript" identifier. Text messages between the various 
participants are transmitted to each other over a communications network, which 
is typically the Internet. As indicated, the chat user interface may include a text 
entry box 112 into which users are able to enter text messages using the remote 
control. After entry of the message, the user typically selects and actuates a send 
button 114 with the remote control to deliver a new message to add to the chat 
transcript. A scroll bar 116 on the edge of the chat user interface may be 
provided to allow the user to scroll up and down to review older messages that 
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may have scrolled out of the chat region. Chat user interface 110 also includes a 
video region 118 for display of a video signal, i.e., a television program which the 
user was viewing prior to entry of the chat room. The chat interface may further 
include a logo region 120 for display of a selected logo which may be associated 
with the chat room. An informational region 122 also may be provided, the 
information region including various data such as a clock 124, a television 
program identifier 126, i.e., a show name, a program schedule time 128 and an 
interactive link 130. Interactive link 130 may be provided to permit a user to jump 
to a website related to the television content being viewed, or receive other 
interactive data related to the television program. For instance, a user may be 
linked to the NBC website during viewing of an NBC show or may receive sports 
scores during a sporting event. 

e. It would have been obvious to one of ordinary skill in the art at the time of 
the invention to modify Rangan's authoring system having data capture and 
synchronization system at the user's end and adapted to receive broadcast data- 
streams from varying sources and combine and synchronize the streams so the 
data from the two different streams may be integrally displayed as authored. 
System 115 has a central processing unit (CPU) 117 that has a cache memory 
and random access memory (RAM). System 115 may be integrated with a 
computer or components thereof, a WEB TV or components thereof, or another 
type of receiving station capable of capturing and displaying broadcast video. 
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f. The motivation would have been for an effective and particularly for a way 
to more efficiently utilize by virtue of the combined video and Internet browsing 
capabilities of the Client premise equipment it is possible to provide functional 
integration between television broadcast viewing and Internet browsing. 

g. Therefore, it would have been obvious to combine these two references of 
Rangan's and Bruck's disclosures in light of providing a system, method and 
apparatus which efficiently integrates broadcast signals and internet browsing, 
For and enhanced interactive viewing of broadcast signals. 

Together Rangan and Bruck disclosed all limitations of Claims 1-43, and hence, 
are rejected under 35 U.S.C. 103(a). 

As to Claims 1, 13, 17 and 21, Rangan teaches A content related information 
provision apparatus that provides related information on contents including reference 
data arranged in time series, the apparatus comprising (as stated in col. 5, lines 53-60, 
and col. 3, lines 53-54, method and apparatus is provided which allows a authoring 
functioning using software modules, in a video editing mode to initiate tracking of any 
image entity or entities in a video data stream which have timing markers inserted 
into the first video data stream): 

related information receiving means that receives related information off contents 
from one or more user devices (as stated in col. 5, lines 64-67, col. 6, lines 1-5, col. 14, 
lines 23-35, FIG. 1 is a block diagram illustrating an authoring station 11 adapted for 
accepting a video data stream 1 6 from a video head end 1 5 as is known in the art. The 
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original video data source 12 may be any recorded or live source and is not limited as to 
protocol. Authoring station 11 includes a display 14 and an object tracking module 13 
which is adapted to automatically track image entities in a video presentation, and to 
synchronize the tracking result with the video data stream. FIG. 8 is a block diagram 
illustrating multiple authoring station architecture for authoring system 51. Authoring 
system 51 comprises a plurality of authoring stations 61a-61d that are adapted for 
tracking image entities and providing additional annotation, including annotation 
regarding tracked entities. An authoring station such as station 61a is analogous to 
authoring station 11 of FIG. 1 except for added capability, for providing further 
annotation other than tracking information and interactive applications as described); 

related information storing means that stores the received related information; 
(as stated in col. 6, lines 52-63, col. 12, lines 20-29, col. 6, lines 6-16, col. 3, lines 65- 
67, col. 4, lines 1-14, suitable computer platform for performing the tracking process 
enabled by tracking module 13 is a PC/VDU running Windows with a central processing 
unit (CPU) operating at least 300 megahertz and having at least 64 megabytes of 
random access memory (RAM). Video frequency or a frame speed of 30 frames per 
second (FPS) is utilized in this example as a suitable and compatible standard for 
processing. The display size of the video picture in this instance is 352.times.240 pixels, 
which is a reduced resolution from that normally displayed in a video presentation on 
most end use stations, but provides a suitable format for the present tracking process. 
Subscriber (end users) to a service specializing in providing video media wherein 
interactive advertisements are presented may obtain CPE equipment adapted to 
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display, identify, and provide, through interactive device, methods for obtaining 
additional information regarding image entities. Such interaction may be a simple 
mouse click on the entity during playback of the video which may invoke a link to a 
network-based data-server that may deliver the data to the end user via modem 
connection or the like. The overall purpose of the authoring station is addition of 
innovative material to the video data stream, such as text overlay, graphic icons and 
logos for advertisement, some of which may be associated with identity and address 
data to allow a viewer at a computerized end station to access advertisements and 
other data which may be associated with individual entities in the video presentation. 
Advertisements may, for example, be associated with a tracked object. Also the text 
annotations could either be set to track along with an object, or appear in a fixed 
position anywhere on the screen, as they are typical in broadcasts today. In some 
embodiments the first data stream is a live video data stream and the second data 
stream is an annotation data stream authored in synchronization with the first data 
stream. The annotation data stream may include tracking data derived from tracking an 
entity in the first data stream. System for synchronizing a first data stream with a 
second data stream is provided, comprising a first controllable dynamic buffer reading 
the first data stream for inserted frame identifiers identifying frames from the second 
data stream to be displayed with frames from the first data stream to accomplish 
synchronization; a second controllable dynamic buffer reading frame identifiers in the 
second data stream; and a control module controlling the dynamic buffers, adjusting the 
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relative position of the two dynamic streams to accomplish synchronization according to 
the data read from the two data streams); 

reference information receiving means that receives reference information from 
the one or more user devices, the reference information specifying contents to be 
referred to by the related information and specifying a data reference position in the 
contents (as stated in col. 11, lines 45-52, col. 13, lines 18-28, col. 14, lines 52-65, col. 
15, lines 5-12, Tracking module 13 set-up includes operations such as pre-selecting 
tracking box shape and size, pre-selecting number of target and test pixels, and so on. 
Process execution encompasses initiating and carrying out the automated tracking 
process including iterative execution of the test algorithm. A data-stream generator 
produces a data-stream with tracking coordinates for each frame interval which is 
synchronous to the frame intervals of video stream. Authoring system 51 receives 
stream 49 for the purpose of authoring the stream as described above. Authoring 
system 51 comprises multiple dedicated authoring stations equipped with software 
capable of tracking images within the video stream and adding annotations including 
interactive icons, text, animated graphics and sounds, as is disclosed in enabling detail 
below. Due to a unique synchronous architecture which allows multiple image tracking 
and annotation operations to be performed simultaneously, the resulting output streams, 
shown here as stream 53 (video stream) and stream 55 (annotation stream). After 
authoring is performed via stations 61a-61d, as illustrated, there are two streams that 
are output and transmitted from each authoring station. One is video stream 53 which in 
most embodiments will remain unchanged under most circumstances. The other is an 
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annotation data stream containing annotations such as are individual to each authoring 
station where annotations are performed. For example, annotation streams 62a-62d are 
associated with stations 61a-61d respectively. Annotation streams 62a-62d differs from 
each other only in that the annotations contained therein are different as a case of being 
authored in different authoring stations. An authoring server 63 is provided and adapted 
to combine annotation streams 61a-d into one annotation stream 55 which is analogous 
to stream 55 of FIG. 7. In this way, all annotations performed separately may be 
combined and may act in unison. Video stream outputs from the separate authoring 
stations converge, or more specifically, remerge into video stream 53 as illustrated via a 
horizontal, left-facing arrow labeled element number 53); 

reference information storing means that stores the received reference 
information; (as stated in col. 15, lines 15-23, If there is a known latency with respect to 
recombining streams 62a-62d in server 63, then video stream 53 must be re- 
synchronized with annotation stream 55 before stream 55 becomes output. In this case, 
stream 53 is diverted over path 65 into server 63, stored in controllable dynamic buffers 
as done in authoring stations and delayed until it is synchronous with stream 55 before 
it exits server 63 over path 67. In this way, streams 55 and 53 remain synchronous on 
output from the authoring server system. In an alternate embodiment, synchronic delay 
may be performed in a separate server (not shown); 

and information delivering means that transmits information for displaying on a 
display regarding which related information is inputted at a specific user device from 
among said one or more user devices, and that delivers at least one of the stored 
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related information or the stored reference information to the one or more user devices; 
(as stated in col. 17, lines 16-32, col. 20, lines 47-59, col. 4, lines 64-67, col. 5, lines 1- 
3, col. 13, lines 56-67, col. 14, lines 1-6, bulk of annotation in the form of inserted text, 
graphical icons, universal resource locators (URL's), interactive shapes, and so on will, 
in many embodiments, be at least partly associated with tracking coordinates of an 
image and therefore will depend on those frame by frame coordinates. For example, an 
interactive icon may follow a moving image entity and be visible by an end user as in 
case of advertisement logos for sponsors of sportspersons in a sporting event. Text 
blocks and the like may take similar association. Hence, the specific content of 
annotations and insertion methods of such annotations may be pre-designed based on 
known facts about the video stream such as what image is to be tracked for what 
advertiser who has what URL's and so on. Execution of those annotations may be 
automatic according to a timed function as described above, or may be performed 
manually, perhaps using a macro or other designed input function. FIG. 11 is a process 
flow chart illustrating logical steps for providing a synchronization signature at the 
authoring end according to an embodiment of the present invention. At step 103 the 
frames of the two streams are identified and monitored as necessary. The software may 
determine, for example, the scope (density) of annotation, the status of available VBI 
and HBI areas, and the frequency of frames for time marking intervals, and so on. This 
step also includes counting frames for the purpose of generating annotation frame 
numbers for signature association purposes. In step 105, serial binary numbers are 
generated in separate sequences that may be used for time marking, physical marking, 
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and frame association. Apparatus and methods are provided allowing data streams to 
be marked while operated in synchronization, and to then be delivered by different 
networks having different latency effects, such that the streams are not synchronous as 
received, but may be re-synchronized using the marks provided while the streams were 
synchronous. Stream 55a, is illustrated as being sent along with stream 53 to 
broadcast system 47 where the streams are combined before broadcasting. In case of 
an analog broadcast the annotation data stream (stream 53) may, for example, be sent 
using the Vertical Blanking Interval (VBI) and synchronized with the video. In case of a 
digital broadcast (using for example MPEG2 digital video format) this annotation stream 
may be sent as a private data stream multiplexed with the audio and video streams in 
the digital broadcast. Annotation stream 55b is alternatively illustrated as being 
delivered over a wide-area-network (WAN) 57, which may be the Internet or other type 
of WAN, directly to one, some or all of end users 48an via a path 59 which represents 
collective individual user WAN connections. Likewise, still other known delivery 
mediums may be used. In some special instances, streams 53 and 55 may be 
combined before leaving authoring system 51. There are many variable possibilities); 

wherein the content referred to in the related information was already distributed 
or will be distributed to the one or more user devices (as stated in col. 12, lines 20-29, 
Subscriber (end users) to a service specializing in providing video media wherein 
interactive advertisements are presented may obtain CPE equipment adapted to 
display, identify, and provide, through interactive device, methods for obtaining 
additional information regarding image entities. Such interaction may be a simple 
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mouse click on the entity during playback of the video which may invoke a link to a 
network-based data-server that may deliver the data to the end user via modem 
connection or the like. 

As to Claims 2, 14, 18 and 22, Rangan teaches content related information 
provision apparatus, method, computer program stored on media and annotation 
system, according to claims 1, 13, 17 and 21, further comprising: 

means that specifies an installation region of a terminal apparatus of the one or 
more user devices to be a delivery destination according to the information delivering 
means (as stated in col. 17, lines 1-5, col. 18, lines 1-5, col. 17, lines 25-32, user 
interface 83 may pre-programmed by an author to supply the appropriate pre-selected 
annotations in a reactive fashion. That is, according to a specific time interval, a signal 
could initiate annotation inserts and so on. Signature application module resident in 
authoring server is initiated after and coordinate tracking and annotation has been 
performed and synchronization of Video data stream and Annotation data stream is 
achieved in the authoring server. Specific content of annotations and insertion methods 
of such annotations may be pre-designed based on known facts about the video stream 
such as what image is to be tracked for what advertiser who has what URL's and so on. 
Execution of those annotations may be automatic according to a timed function, or may 
be performed manually, perhaps using a macro or other designed input function); 

and information changing means that changes contents of related information 
and/or reference information, which should be delivered, according to the installation 
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region (as stated in col. 18, lines 29-32, col. 18, lines 6-10, and col. 17, lines 55-60, 
number/time marker-generator module generates code to represent frames in 
annotation stream and also to represent time markers in video streams. Video and 
annotation streams are given frame-specific identification and marking as well as 
time-stamped so that they may later be synchronized by using inserted data 
corresponding to the frame-specific identification. The signature streams are then 
sent to their respective broadcast and/or data-transmission systems to be sent to an 
end user). 

As to Claims 3, 15, 19 and 23, Rangan teaches content related information 
provision apparatus, method, computer program stored on media and annotation 
system, according to claims 1, 13, 17 and 21, wherein the contents refer to a broadcast 
program, and the reference information includes information specifying a broadcasting 
station, which broadcasts or has broadcasted a program, and information specifying a 
date and time when a reference part in a program is broadcasted or has been 
broadcasted (as stated in col. 6, lines 39-50, col. 3, lines 33-38, authoring station 
comprises software module that may process various media analog or digital not 
limited to common formats such as Audio Video Interleave AVI and Moving Pictures 
Experts Group MPEG. Video source may embody any video source (contents) that 
may be known in the art such as a CD-ROM, Satellite TV, cable TV, VCR, Internet 
Server. Video source may provide prerecorded video or live broadcast video. The 
first data stream is a live video data stream and timing markers are placed by the 
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writer in the first data stream, the second data stream is an annotation data stream 
(reference information) authored in synchronization with the first data stream. The 
annotation data stream includes tracking data with time stamps derived from 
tracking an entity in the first data stream (contents)). 

As to Claims 4, 5, 16, 20 and 24, Rangan teaches content related information 
provision apparatus, method, computer program stored on media and annotation 
system, according to claims 3, 14, 18 and 23, wherein the reference information 
specifies a broadcasting station, which broadcasts or has broadcasted a program, using 
a channel number (as stated in col. 6, lines 39-50, col. 3, lines 33-38, col. 9, lines 49-52 
col. 7, lines 34-41, col. 1, lines 40-55, first data stream is a live video data stream 
(broadcasted program) and frame identifiers and timing markers are implemented 
by various methods, such as binary numbers and are placed by the writer in the first 
data stream, timing marks in some cases are at intervals of a number of frames by 
convention and are representation of value table for storing values. The values being 
collected and stored may be kept in any logical order such as in a data list or the like. 
The second data stream is an annotation data stream (reference information) 
authored in synchronization with the first data stream. The annotation data stream 
includes tracking data (reference information) derived from tracking an entity in the 
first data stream (content). User interaction with such an image entity during viewing 
of a video can be programmed to provide additional network-stored information 
about that entity to suitable customer premises equipment adapted to receive and 
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display the information, as an overlay on the display of the dynamic video containing 
the subject image entity. In some systems computer elements, such as a CPU, 
memory, and the like, are built into the familiar chassis of a TV set. In such a system, 
the TV screen becomes the display monitor in the computer mode. In such a system, 
conventional TV elements and circuitry are incorporated along with the computer 
elements, and capability is provided for a user to switch modes, or to view recorded or 
broadcast video with added computer interaction. One may thus, with a properly 
equipped system, select to view analog TV programs, digital TV programs, conventional 
cable TV, satellite TV, pay TV from various sources, and browse the WWW as well, 
displaying WEB pages and interacting with on-screen fields and relational systems for 
jumping to related information, databases, and other WEB pages. The capabilities are 
often integrated into a single display, that is, one may view a broadcast presentation 
and also have a window on the display for WEB interaction). 

As to Claims 6, and 27, Rangan teaches content related information provision 
apparatus, and annotation system, according to claims 1, and 21, wherein the 
information delivering means includes identification information of a site handling 
information resources related to contents in the reference information and delivers the 
identification information (as stated in col. 13, lines 45-48, col. 15, lines 5-8, col. 16, 
lines 27-35, Authoring system comprises multiple authoring stations (site handling 
information) equipped with software capable of tracking images through tracking 
module and provide frame by frame tracking coordinates and to be a vehicle through 
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which additional annotations may be provided through user interface, an author may 
set up the parameters for tracking, with reference and add additional annotation such 
as static or moving image icons, formatted text, animated graphics, to a single live or 
pre-recorded video feed via an innovative synchronous architecture which allows 
multiple image tracking and annotation operations to be performed simultaneously in 
near real-time such that delay of the broadcast video to an end user is negligible. 
Authoring server is provided and adapted to combine annotation streams through 
frame-specific identification numbers, time stamps, and markings, into one 
annotation stream, in this way, all annotations performed separately may be combined 
and may act in unison at the users end). 

As to Claims 7, and 28, Rangan teaches content related information provision 
apparatus, and annotation system, according to claims 1, and 21, wherein the 
information delivering means includes information for correcting deviation of a clock in a 
terminal apparatus to be a delivery destination in the reference information and delivers 
the information (as stated in col. 18, lines 30-33, col. 19, line 67, and col. 19, lines 51- 
54, more than one, or several, of the synchronization schemes may be used in any 
instance. A number/time marker-generator module generates code to represent 
frames in annotation stream and also to represent time markers in video stream. In 
some instances it may be necessary for the servers and the user's equipment to be 
synchronized in time, the clock at the server end is synchronized and checked with, 
and corresponded clock at the user's end). 
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As to Claims 8, and 29, Rangan teaches content related information provision 
apparatus, and annotation system, according to claims 1, and 21, wherein the 
information delivering means includes a characteristic amount of contents at a reference 
position in the contents in the reference information and delivers the characteristic 
amount (as stated in col. 16, lines 27-34, tracking module is provided and adapted to 
track an image and provide frame by frame tracking coordinates and to be a vehicle 
through which additional annotations may be provided through user interface, an 
author may set up the parameters for tracking such as are described with reference to 
specific position, as well as add additional annotation such as static or moving image 
icons, formatted text, animated graphics, sounds and the like). 

As to Claims 9, and 30, Rangan teaches content related information provision 
apparatus, and annotation system, according to claims 1, and 21, wherein the 
information delivering means delivers plural pieces of reference information collectively 
(as stated in col. 17, lines 41-45, synchronization module is added in authoring 
server and adapted to synchronize separate (plural) annotation streams (reference 
information) before combining them and synchronizing them with the output video 
stream). 

As to Claims 10, and 33, Rangan teaches content related information provision 
apparatus, and annotation system, according to claims 1, and 21, wherein the 
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information delivering means delivers the related information and/or the reference 
information in accordance with an HTTP (Hyper Text Transfer Protocol) (as stated in 
col. 1, lines 35-38, and lines 47-55, developers are introducing integrated systems 
combining TVs with computer subsystems, so a TV may be used as a WEB 
browser, (viewing web pages in HTTP) or a PC may be used for enhanced TV 
viewing (broadcast programs). One may thus, with a properly equipped system, select 
to view analog TV programs, digital TV programs, conventional cable TV, satellite TV, 
pay TV from various sources, and browse the WWW as well, displaying WEB pages 
and interacting with on-screen fields and relational systems for jumping to related 
information, databases, and other WEB pages. The capabilities are often integrated 
into a single display, that is, one may view a broadcast presentation and also have a 
window on the display for WEB interaction). 

As to Claims 11, and 34, Rangan teaches content related information provision 
apparatus, and annotation system, according to claims 1, and 21, wherein the 
information delivering means delivers the related information and/or the reference 
information in accordance with an SMTP (Simple Mail Transfer Protocol) (as stated in 
col. 5, lines 66-67, col. 6, lines 1-5, lines 35-38, col. 14, lines 20-22, original video data 
source may be any recorded or live broadcast source and is not limited as to protocol. 
Annotations and alterations may be made by adding annotations including 
interactive icons, text, animated graphics and sounds, to the input data stream 
source in the authoring station, and output from authoring station is meant to be a 
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indication of data output, and not to indicate that there is a single data stream. There 
may be multiple streams of various protocols and is not limited (any protocol may 
be used) by the medium of transport protocols used for transmitting the video and 
the annotation data stream). 

As to Claims 12, and 35, Rangan teaches content related information provision 
apparatus, and annotation system, according to claims 1, and 21, wherein the 
information delivering means designates a character string, which is capable of 
identifying reference information, in a header of a delivery message at the time of 
delivery of reference information (as stated in col. 18, lines 35-40, lines 29-34, col. 19, 
lines 61-65 and col. 20, lines 60-65, user interface may pre-programmed by an 
author to supply the appropriate preselected annotations in a reactive fashion. That 
is, according to a specific time interval, a signal could initiate annotation inserts and 
so on. In other embodiments, an author may physically enter an annotation via 
pressing pre-defined keys on a keyboard and so on. There are many known methods 
for inserting annotations. Several separate signature methods, for analog video, digital 
video, are used to insure that synchronization information survives to the customer's 
equipment end. A number/time marker-generator module generates code (string) 
to represent frames in annotation stream and also to represent time markers in video 
stream. Further, if the video is in a digital format, such as Motion Picture Expert's Group 
(MPEG), SMPTE-like time stamps may be inserted into the headers in the data 
packets which are then compared with time stamps in the control stream. Annotation 
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frame numbers are written into areas associated with video frames as well as to the 
appropriate annotation frame headers). 

As to Claims 25-26, Rangan teaches annotation system, according to claim 21, 
wherein the transmitter is further configured to transmit a name of a bulletin board, in 
which a corresponding remark is written, with the name included in the reference 
information (as stated in col. 14, lines 35-40, and col. 16, lines 51-60, FIG. 8 is a block 
diagram illustrating a multiple authoring station architecture for authoring system that 
are adapted for tracking image entities and providing additional annotation, add 
annotation such as static or moving image icons, formatted text, animated graphics, 
sounds including annotation regarding tracked entities. An annotation manager 
converts annotation data, input during annotation processes and the data relating to the 
tracked entities output from the tracking module, to metadata for more compact 
transmission in output data stream containing information about the various 
annotations added by the author of authoring station and the tracking co-ordinates 
of the tracked entities and is analogous to the annotation stream (reference 
information). User interface provides considerable option and capability for entering 
commands to add image icons, animated graphics, following tracked objects or static or 
moving independently in the video in predefined manner, formatted text captions and 
so on). 
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As to Claim 31, Rangan teaches annotation system, according to claim 21, 
further comprising: 

a unit that, in writing of a remark, transmits an execution code for automatically 
acquiring reference information or urging a user to input a remark to a terminal 
apparatus side of a requiring source of the one or more user devices (as stated in col. 
14, lines 35-40, single annotated video-stream is output from display module to a 
suitable connected display monitor or screen. An input signal represents user 
interaction with an entity in video stream as it is displayed. Such an input signal may 
trigger downloading of additional detailed information regarding the subject of 
interaction. Interaction signal results from a mouse click or other input command such 
as may be initiated via a connected keyboard or a remote pointing device or the like). 

As to Claim 32, Rangan teaches annotation system, according to claim 21, 
further comprising: 

a unit that, concerning an execution code for automatically acquiring reference 
information stored in the terminal apparatus of the requesting source of the one or more 
user devices in advance or urging a user to input a remark, transmits information 
necessary for starting the execution code to the terminal apparatus (as stated in col. 17, 
lines 58-65, video/data stream signature operation is executed after coordinate 
tracking and annotation operations are performed in an authoring station. The 
signature streams are then sent to their respective broadcast and/or data- 
transmission systems to transmit information to an end user. Video/annotation 
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stream capture and synchronization operations, executed via software on customer 
premises equipment is executed at the user's end for a single combined stream to be 
viewed by the user). 

As to Claim 36, Rangan teaches annotation system, according to claim 23, 
further comprising: 

a specification unit that, concerning a program series to be an object of an 
argument in a bulletin board, specifies a broadcast schedule for the next broadcast of 
the series (as stated in col. 6, lines 6-16, Purpose of the authoring station is addition 
of innovative material as annotations and alterations to the video data stream, such 
as text overlay, graphic icons and logos for advertisement and interaction with 
viewer, associated with identity and address data to allow a viewer at a computerized 
end station to access advertisements and other data which may be associated with 
individual entities in the video presentation); 

and a unit that transmits the broadcast schedule to a terminal apparatus of a 
request source of the one or more user devices (as stated in col. 7, lines 29-40, 
Through tracking process and additional editing processes a moving region associated 
with the image entity in a display may be made to be interactive and identifiable to an 
end user. User interaction with such an image entity during viewing of a video can be 
programmed to provide additional network-stored information about that entity to 
customer premises equipment adapted to receive and display the information. Such 
network-stored information (broadcast schedule) is transmitted and may be 
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displayed, as an overlay on the display of the dynamic video containing the subject 
image entity). 

As to Claim 37, Rangan teaches annotation system, according to claim 36, 
further comprising: 

a setting unit that sets a bulletin board for each program series and performs 
download of the program schedule from a screen displaying information in the bulletin 
board or a screen displaying a list of remarks in the bulletin board (as stated in col. 22, 
lines 34-42, single annotated video-stream is output from display module to a suitable 
connected display monitor or screen. An input signal represents user interaction with 
an entity in video stream as it is displayed. Such input signal may trigger 
downloading of additional detailed information of network-stored information 
(broadcast schedule) regarding the subject of interaction. Interaction signal results 
from a mouse click or other input command such as may be initiated via a connected 
keyboard or a remote pointing device or the like). 

As to Claim 38, Rangan teaches annotation system, according to claim 23, 
further comprising: 

a specification unit a rebroadcast schedule for a program to be an object of a 
remark (as stated in col. 23, lines 32-40, user or users that receive the video data via 
broadcast, and the annotation data via a WAN, or the Internet. Additional data, 
network-stored information (rebroadcast schedule) obtained by a user through 
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interaction with a tracked entity in the video may be personalized and specific to the 
user through annotation data); 

and a unit that transmits the rebroadcast schedule to a terminal apparatus of a 
request source of the one or more user devices (as stated in col. 23, lines 32-40, col. 
22, lines 57-67, In a case such as this a user would, perhaps, obtain a subscription to 
the service (program, broadcast), for other rebroadcast schedule and data delivery 
methods. Prerecorded and authored video feed from a source connected to an 
optional input module may be synchronized with a previously stored and annotated data 
stream from a source connected to a second optional input module as long as the 
signature process was applied to both streams. Interaction with tracked entities and the 
like associated with the prerecorded streams may be sent to a participating Internet 
server or the like for rebroadcast through the modem sub-module and the system is 
on-line during viewing). 

As to Claim 39, Rangan teaches annotation system, according to claim 38, 
wherein the transmitting means transmits the rebroadcast schedule with the 
rebroadcast schedule included in reference information (as stated in col. 23, lines 32- 
40, user or users that receive the video data via broadcast, and the annotation data 
via a WAN, or the Internet. Additional data, network-stored information 
(rebroadcast schedule) obtained by a user through interaction with a tracked entity in 
the video may be personalized and specific to the user through annotation data 
(reference information)). 
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As to Claim 40, Rangan teaches annotation system, according to claim 21, 
further comprising: 

a designate unit configured to designate retrieval conditions for a remark (as 
stated in col. 22, lines 34-42, single annotated video-stream is output from display 
module to a suitable connected display monitor or screen. An input signal represents 
user's interaction with an entity in video stream as it is displayed. Such a signal may 
trigger downloading of additional detailed information regarding the subject of 
interaction), 

and retrieving unit configured to retrieve a remark across plural bulletin boards 
on the basis of the designated retrieval conditions (as stated in col. 22, lines 34-42, 
Interaction signal results from a mouse click or other input command such as may 
be initiated via a connected keyboard or a remote pointing device or the like by the 
user's). 

As to Claim 41, Rangan teaches annotation system, according to claim 40, 
wherein the bulletin board system uses a keyword included in a remark or designated 
separately at the time of writing the remark as retrieval conditions (as stated in col. 22, 
lines 34-42, Interaction signal results from a mouse click or other input command 
such as may be initiated via a connected keyboard or a remote pointing device or 
the like by the user's). 
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As to Claim 42, Rangan teaches annotation system, according to claim 40, 
wherein the bulletin board system uses a name or an ID of a user who has written the 
remark as retrieval conditions (as stated in col. 7, lines 29-40, Through tracking 
process and additional editing processes a moving region associated with the image 
entity in a display may be made to be interactive and identifiable to an end user. 
User interaction with such an image entity during viewing of a video can be 
programmed to provide additional network-stored information about that entity to 
customer premises equipment adapted to receive and display the information. Such 
network-stored information (broadcast schedule) is transmitted and may be 
displayed, as an overlay on the display of the dynamic video containing the subject 
image entity). 

As to Claim 43, Rangan teaches annotation system, according to claim 40, 
wherein the bulletin board system uses a date and time when the remark is written as 
retrieval conditions (as stated in col. 18, lines 29-34, number/time marker-generator 
module generates code to represent frames in annotation stream and represent time 
markers in video stream). 

Response to Arguments 

4. Applicant's arguments filed on 06/30/2008 have been fully considered, but they 
are not persuasive. 
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a. Applicant's arguments with respect to Claims 1-43 have been considered 
but are moot in view of the new ground(s) of rejection. 

b. Applicant argues that the applied references do not suggest the feature of: 

c. "related information receiver that receives related information on contents 
from one or more user devices" 

Rangan and Bruck both disclose system, method and apparatus for 
authoring stations and set-up boxes having receiver and transmitter for receiving 
both the broadcast signals and data signals arriving through different medium, 
such that they have the benefit of both broadcast signals and internet browsing, 
and they disclose all the features and elements of the present application. 

Therefore, Applicant's arguments are not persuasive regarding all these 
features, elements of the present application. 

Hence Examiner respectfully disagrees with Applicant's arguments on 
page 15-19, and maintains his rejection. 

Action Final 

5. Applicant's arguments necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
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TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 



Conclusion 

6. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Muktesh G. Gupta whose telephone number is 571-270- 
5011. The examiner can normally be reached on Monday-Friday, 8:00 a.m. -5:00 p.m., 
EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, William C. Vaughn can be reached on 571-272-3922. The fax phone 
number for the organization where this application or proceeding is assigned is 571- 
273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
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Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

MG 

/William C. Vaughn, Jr./ 

Supervisory Patent Examiner, Art Unit 2144 



